Empirical studies on weight noise injection based online learning algorithms

While weight noise injection during training has been adopted in attaining fault tolerant neural networks (NNs), theoretical and empirical studies on the online algorithms developed based on these strategies have yet to be complete. In this paper, we present results on two important aspects in online learning algorithms based on combining weight noise injection and weight decay. Through intensive computer simulations, the convergence behaviors of those algorithms and the performance of the NNs generated by these algorithms are elucidated. It is found that (i) the online learning algorithm based on purely multiplicative weight noise injection does not converge, (ii) the algorithms combining weight noise injection and weight decay exhibit better convergence behaviors than their pure weight noise injection counterparts, and (iii) the neural networks attained by those algorithms combining weight noise injection and weight decay show better fault tolerance abilities than the neural networks attained by the pure weight noise injection-based algorithms. These empirical results not just can supplement our recent work done on the convergence behaviors of the weight noise injection-based learning algorithms [16], but also provide new information on effect of weight noise and weight decay on the neural networks that are generated by these algorithms. Keywords-Cross Entropy Error, MLP, Mean Square Errors, Regression, Weight Decay, Weight Noise Injection